Splitting rate matrix as a definition of time reversal in master equation systems 
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Motivated by recent progresses in nonequilibrium Fluctuation Relations, we present a generalized 
time reversal for stochastic master equation systems with discrete states that is defined as a splitting 
of the rate matrix into irreversible and reversible parts. An immediate advantage of this definition 
is that a variety of fluctuation relations can be attributed to different matrix splitting. Additionally, 
we also find that, the accustomed total entropy production formula and conditions of the detailed 
balance must be modified appropriately to account for the presence of the reversible part, which 
was completely ignored in the past a long time. 
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Fluctuation theorems or fluctuation relations [11-1141] are a variety of exact equalities about statistics of entropy 
production or dissipated work that are held even in far from equilibrium regimes. In near-equilibrium region, these 
relations reduce to the famous fluctuation-dissipation theorems (FDTs) pi Irl fl5l - [l7j . The discovery of these fluctuation 
relations significantly advances our understanding about nonequilibrium physics, and particularly about the second 
^ ' law of thermodynamics of small systems fl8j . 

The fluctuation relations are very relevant with the concept of time reversal [34131 [l9l . [20j . For instance, under 
■ the framework of Markovian stochastic systems, previous work has proved that a majority of them can be derived by 
^j" 1 | a ratio of the probability densities of observing a trajectory in a original system and the reversed trajectory in the 
time-reversed system. Very recently, Chetrite and Gawedzki [2l[ further elaborated this observation and presented a 
generalized time reversal definition on continuous diffusion processes. Different from conventional definition of time 
O . reversal as simply changing time parameter in a stochastic dynamics into minus, they explicitly defined time reversal 
i ^ i ' as a splitting of drift vector into irreversible and reversible parts which possess distinct rules under the transformation 
of t—¥—t. Because of freedom of the split ting , a variety of time reversal and corresponding fluctuation relations are 
s ! ' obtained, e.g., the Hatano-Sasa equality [llj arising from a novel time reversal with nonzero reversible drift. The 
^ \ importance of the generalized time reversal was shown again when we understood the origin of a generalized integral 
fluctuation relation (GIFR) on general diffusion processes [22l.l23l]. 

In addition to the continuous diffusion process, another typical and important Markovian process is master equation 
ly-j . with discrete states and continuous time (24|. Although the latter is more general than the former in principle, due 
to their highly formal analogy, many results and evaluations about the fluctuation relations in the diffusion processes 
could be established correspondingly in the master equation systems (rj [l3l [2oT - |28| . Nevertheless, whether a similar 
time reversal definition exists and how to define it are not investigated yet. We must emphasize that it is not trivial as 
one might think of at first glance. Except for positive transition rates, in these master equation systems there are not 
quantities such as drift vector and diffusion matrix that have intuitive rules of transformation as time reversed [29| . 
Somewhat surprisingly, here we will show that an physically relevant answer indeed exists. 

The organization of this work is as follows. In sec. [TT1 we briefly review a GIFR in the master equation systems 
that we found very recently (2?]|. The reason we use the GIFR rather than other famous fluctuation relations is its 
generality. Additionally, the necessary of extending the conventional time reversal will be brought forth naturally in 
deriving the GIFR. In order to interpret an unknown matrix in this relation, in sec. IHII we present a generalized time 
reversal in these systems as a splitting of rate matrix into reversal and irreversible parts. The consequences of this 
new definition will be discussed in sec. IIV1 which includes reinvestigation of the fluctuation relations from a point of 
view of the splitting, generalization of total entropy production and conditions of the detailed balance. Section fVl is 
the summary. 
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II. REVIEW OF THE GIFR IN THE MASTER EQUATION SYSTEMS 



Assume a Markovian process with discrete states and continuous time is described by a master equation 27 

dp n (t) 



dt 



[H(t)p(i)] 



(1) 



where n is the state index which may be a vector, the iV-dimensional column vector p(£)=(pi, • • • ,pat) t is the probabil- 
ity of the system at individual states at time t, and the matrix element (H) mn =H mn >0 (m^n) is the time dependent 
or independent rate and (H)„„=— ^2 m ^ n H mn . Given a normalized positive column vector f(f)=(/i, ■ • ■ , Jn) T and a 
NxN matrix A whose elements (A) mn —A mn (m ^ n) satisfy condition H mn f n +A mn > and A nn —~ Y^mjtn A m n, we 
found that the inner product f T (i')v(i') (t'<t) is i'-invariable if the vector v(i') = (ui, • • ■ , vn) t satisfies a perturbed 
backward equation 



dv n {t') 

dt' 



- (H T v)„ 

- /- 1 (^f-Hf)„« n + /- 1 [(Al), 



(A T v)„], 



(2) 



where the final condition v n (t)=d n , and iV-dimcnsion column vector 1 = (1, • ■ ■ , 1) T . This is easily proved by noticing 
the time differential property and the transpose property of a matrix. Employing the Fcynman-Kac and Girsanov 
formulas for discrete jump processes [13] , the solution of ([2|) has a path integral representation given by 



iY 



£ f m (Or- (e'SoJ^(r))dr dx{t)) = f T (t)d 



(3) 



and the integrant in the functional is 



(r) 



-0 T f+Hf+ Al] 



x(r) 



2[/x(r)A] 



with 



Q[B] 



--B x (t)x(t) _ hi 



B 



x(r+ )x(r — ) 



r-)(r) 



(r+)x(r-) 



-)(r) 



(4) 



(5) 



where m,0 ( ) is the expectation over all trajectories x generated from the system {T]) with a fixed state m at initial 
time 0, x(t) is the discrete state of the system at instant time r, x(r~) and x(t + ) represent the states just before 
and after a jump occurring at time r, respectively, and we have assumed the jumps occur k times for a trajectory. 
By selecting different f and A, the GIFR may be reduced into different fluctuation relations in the literature [271 ] . 
Although ([2]) seems complicated, in fact it could be arranged into a concise form: 



dqn(s) 

ds 



[H(.s)q( S )] fi , 



where the elements of the probability q(s) are 

q n {s) = [f(i) T d]-7»(*>n(£') 

with s=t—t', and the elements of the matrix H(s) are 

Hnrh(s) = f m \t') [H mn (t')f n (t') + A mn {t')\ 



(6) 



(7) 



(8) 



for m^n and Hmf n (s)=— X^n^m Hnm{s), and n represents an index whose components are the same or the minus of 
themselves depending on whether they are even or odd under time reversal t— >— t. Considering that the parameter s is 
analogous to a reversed time and Hnm(s)=H nm (t') as specifically selecting A whose elements are the probability flux 
[J(t)}mn=(H nm f m —H mn f n ), we simply name ^ as generalized time reversal of the original system (fT]). However, 
why A is almost arbitrary instead of equaling probability flux only was not fully understood previously. Hence, 
defining the generalized time reversal from a different point of view seems very essential. 



3 



III. SPLITTING RATE MATRIX AS TIME REVERSAL 

Let us begin with two general matrix identities: 

[M(a.b)] n -[Ma] n 6„ + a„[M T b] n = [S T b]„ - [Sl]„6 n , (9) 
[M(a.b)] n + [Ma] n 6„-a„[M T b] n = [J T b]„ - [Jl] n 6„, (10) 

where both a and b arc A^-dimensional vectors, M is NxN matrix with J2 m M mn =0, (a.b)=(ai&i, • • ■ ,ajs[bN) T 
represents an array multiplication of the vectors, and the matrixes S and J are constructed by M and a: 

(S)mn — ^nm^ ~t~ M mn Cl n : (J)mn — Mum^m ^mn^n- (H) 

They are symmetric and antisymmetric, respectively. Proving ([9]) and (|10|) is straightforward. 

Now we assume that the rate matrix can be splittcd into a sum of "reversible" and "irreversible" matrixes, namely 
H=H I0V +H lrr . Particularly, we require H 1 1 ^ n >H' l ^ l (m^n), the reason of which will be seen shortly. It is easily to 
see that H*" n is always greater than zero for H mn >0, though the sign of H^ n is indefinite. We further assume H is 
transformed into H=H rcv +H nl under time reversal and 

H fiih( s ) = Cnf*'). Hfhl( s ) = -^n(0- ( 12 ) 

Obviously, time-reversed matrix H still possesses rate interpretation. Using these definitions and identities we rewrite 
the right-hand side of dU as 

[Hqk = — ^{[H irr (f.v)] fi + [H rev (f.v)] fi } (13) 



where 



-[(J m -S rev ) 1 v]„} ) 



(S ICV ),„„ = H^f m + H™lf n , (14) 

(J )mn ~ H nm f m — H mn f n . (15) 



Comparing (|13j) with ([2]), we immediately find that 

[A(f)] mn = (J irr ) mn - (S rcv ) m „ (16) 
= [J(f)] mn -2H!Zfm, (17) 

(|17|) implies why the matrix A is almost arbitrary: the reversible matrix H rcv to be determined is responsible for this 
freedom. In addition, we also notice that the condition of H mn f n +A mn >0 mentioned in last section is identical with 
H% n >HlZ here. 

In practice, we may prior know the matrix A or time-reversed rate matrix H. Under these circumstances, the 
splitting can be constructed conversely as 

^nm — (^"nm fm Hjnnfn -^-rnn^) / ^ f m i (1^) 



H^Kt') = [H mn (t') - Hj lA (s)}/2, (20) 
H% n (*') = [H mn (*') + H r%fl (s)} /2. (21) 

Regardless of how a split ting of the rate matrix is achieved, substituting (fTBj) into the integrand (Q| we can obtain its 
new expression given by |35| 



J[f)H -,H rev ] = -Aln/ x(T) (r) + J[l,ff",ir], (22) 
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where the second term on the right-hand side is 



rrirr rrrev h 

m^x(r) x(r+)x(r ) x(r+)x(r ) j=l 

([22]) is a consequence of the arbitrariness of f in , a brief explanation of which is given in Appendix A. Noting (|23|) 
also satisfies a fluctuation relation if the initial distribution of the system is uniform. (|2"21 and (|23[) are the central 
results of this work. 

IV. DISCUSSION 

Because the physical interpretation of the GIFR ([3]), its connection with the other fluctuation relations, and its 
detailed version have been partially investigated [27| . in the remainder we only present several new results obtained 
from a point of view of the splitting. 

A. Total entropy production rate 

If f is the system's probability vector p and the splitting is prior known from physical consideration, e.g., time 
reversibility below, (|22[) is the balance equation of trajectory entropy: the first term on the right-hand side is the 
change in trajectory system entropy, the second term is the change in trajectory environmental entropy along a 
specified trajectory, and the left-hand side is the total trajectory entropy. This interpretation has been widely 
accepted [TH, HjJ H|| [H, [3(| ■ To our knowledge, however, the expression of the trajectory environmental entropy (|2"3"]l 
that involves both even and odd discrete variables is firstly given here. This new formula also reminds us that the 
total entropy production for the master equation systems with irreversible and reversible rate matrixes is 

(j)= 2EE^o>- 

m^n n 

( rrirr rrrcv \ „ 

- e E( g ™ + H ™)pm in r z:\ Pn > o. (2 4) 

The inequality may be proved easily by using (x—l)>lax or using the Jensen inequality for the GIFR ([3]). The 
expression of (|24[) is significantly distinct from the classical entropy production formula given by Schnakenberg (3~l| 
that involves only even variables. 

B. Conditions of the detailed balance 

For a time- independent master equation system that has equilibrium state p eq , we may physically require it invari- 
able under time-reversal, namely, H=H. According to (f2"0)) and (|21[) an unique splitting is then 

H m n = \Hmn ~ -^mfi)/2, H mn = (H mn + -ffmfi)/2. (25) 

Under this circumstance the irreversible and reversible parts have distinctive properties: 

rrrev _ rrrcv f7 irr Pf' rr !0f\\ 

Obviously, if the discrete master equation system involves only even states, which is exclusively the object investigated 
in numerous references [241. [29l. |32|. 1331] . the reversible rate matrix must vanish. Because the entropy production is 
zero when the system is in equilibrium state, according to (|24p . all terms on its right hand side must vanish. Hence 
we obtain conditions of detailed balance on the rate matrix and the state which are respectively 

(S rev ) mn = H™X + CW = 0, (27) 
(J )mn = H nm p^ — H mn p 7 ^ = 0, (28) 
E H ™ = 0. (29) 
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We must emphasize that, except for the last equation ([29]) that is from the steady-state requirement and was ignored 
in previous literature [24 . l29l l32l [33l ] , the former two equations are fully equivalent with the conventional detailed 
balance condition 

HnmPm = ^mnPraS (30) 

which may be easily checked using (|25p . We see an unusual side of a time-reversible master equation system with 
nonvanishing H rov : the probability flux (J) mn between two states m and n may be not zero even if the system is in an 
equilibrium state; both the irreversible and reversible rate matrixes have contributes to the total entropy production. 
The latter point can be seen more clearly when we rewrite (|24p as 



ZTirr „ 

mi tn n n nm p m 

2^U nm p m ln{-—— ). 

by taking (|27j) - ((29|) into account. To our best knowledge, there are very fewer stochastic jump processes with both 
even and odd variables in the literature. In Appendix B we present a simple mathematical model to exemplify their 
intriguing features. 

C. Jarzynski and Hatano-Sasa equalities 

For a time-dependent master equation system, we may select the simplest case with A=0. Then (fT8|) and (|T9]) 
become 

H-nm = TTc \Hnmfm ~ H mn f n ), (32) 

H-nm = 177 {Hnmfm + H mn f n ). (33) 

We used a new notation TL instead of H to indicate the speciality of this splitting. Under this case, (f2"2"j) is 

J[{,A = 0] = -Aln/ x(T) (r) + Jll.r.HH, (34) 



and 



1 



/x(r) 



J r [l,H I ",'H rcv ] — — — Px(T)m/(B"ffroi(T)/i(T)] 



_ In £*(,-_,..). (35 ) 

If one further selects f to be the instant equilibrium solution p cq (t) or the instant nonequilibrium steady-state solution 
p ss (t) of the master equation if it has, the GIFR is respectively the famous Jarzynski equality about dissipated 
work 0, Q or Hatano-Sasa equality about excess entropy Noting the first term of the right-hand side of the 

above equation in both cases vanishes because of the steady-state condition. It is worth pointing out that for the former 
equality, the detailed balance conditions (|27|) and (|28|) imply that such a splitting is trivial because of 'H^ l (t)=H^ l (t) 
and H 1 ^ n (t)=H^(t), and ([33)1 has other three different but equivalent expressions: 

„eq fjirr rrrev 

,„ P x(r-) _ Ji x(r-)x(r+) _. , Jj x(r-)x(r+) 1 

111 cq — m . rr ni[ y j 

^x(t + ) w x(t+)x(t-) x(t+)x(i—) 

j^"irr jLfrev 

_ j n x(r-)x(r+) x(r~)x(r+) ,„g\ 

rrirr _i_ tticv ^ ' 

j x(t+)x(t") t "x(t+)x(t-) 

On the other hand, if A1=0 the integrand ((4]) can be decomposed into 

J[f,A}= J[f,0] + Q[/-^A]. (37) 
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This relationship is very intriguing. In addition that the three terms above satisfy integral fluctuation relation [27j 
simultaneously, all of them have important physical implications in the nonequilibrium steady-state thermodynam- 
ics [U ED, 0, Hil HH . Here we do not repeat previous interpretations but present a simple understanding from a 
point of view of the splitting. For a nonequilibrium master equation system with H(t)=H lrr (i) and assuming it has 
unique steady-state as external time-dependent parameter fixed, we have 

J[p,J(p)]= - -fln^P + J[p ss ,J(p ss )] (38) 

aT Px(r) 

= — m{^ + ^o] + e[^] (39) 

Px(r) Px(t) 

= -^lnp x(r) + ^[l,H te ,^ cv ] + Q[^ i ], (40) 
dr j£ (t) 

where (f2"2")l . (f3~4"| . (|3T|) and steady-state condition J(p ss )l=0 are used. ([31))) implies that trajectory total ent ropy 
is the sum of trajectory relative entropy, trajectory excess entropy [ll|, and trajectory housekeeping entropy [14| . 
Particularly, the sum of the first two terms in ([4*U| that was called as nonadiabatic trajectory entropy [28| is just 
J\p, H m , % rcv ], the ensemble average of which is 



(j\p, n hl \ w cv }) = - EE h ™p™ ln ^ Z o (4i) 

m^n n P ™ Pm 

according to ([21]) . In Ref. [30| this quantity was also named as free energy dissipation. 



V. CONCLUSION 



Motivated by the important idea of defining time reversal as a splitting of drift vector in continuous diffusion 
process, we present the same effort in the master equation system with discrete states. Different from the former, we 
define the time reversal in this system as a splitting of the rate matrix into irreversible and reversible parts. Even 
that we find very analogous formulas and results are revealed in these two systems, e.g. (|23p corresponding (7.6) in 
the work of Chetrite and Gawedzki [2l[ or (28) in the work of us |2j|. The advantages of introducing this definition 
are obvious. First, we explain the origin of the matrix A in the GIFR, which was somewhat mysterious to us before 
starting this work. Second, a variety of fluctuation relations in the master equation systems are unified into various 
rate matrix splitting. This point was not acknowledged previously. Additionally, the relationships among these 
fluctuation relations become very clear from this splitting viewpoint, e.g. (|39[) and (|40p . Finally, this definition also 
reminds us the importance of the reversible part of the rate matrix. For instance, the expression of the total entropy 
production must be modified appropriately. To our knowledge, its existence and implications in the master equation 
systems were almost completely ignored for a very long time. 

This work was supported in part by State Key Laboratory of Software Development Environment SKLSDE-2011ZX- 
20 and the National Science Foundation of China under Grant No. 11174025. 



Appendix A: Derivation of (f22j) and ([23]) 

For simplicity in notations, we study a specific case of ([5]) with a final condition is d n =l. Under this circumstance 
the perturbed backward master equation has the time-reversal explanation ([6]) and 

qn[s) = fn(t')v n [t'). (A42) 

Noting that the initial condition of is now f n (t). It is worth emphasizing that we can replace the vector f above 
by other normalized positive vectors, e.g. c=(l/N, • • • , 1/iV). Then we have a new relation 

qa(s) = c n u n (t'), (A43) 

where u n satisfies the same ([2]) except that /„ therein including those in the matrix A(f ) (|16j) are substituted by c n 
and the finial condition becomes u n (t)=f n (t)/c n . According to (|A42j) . ()A43j) . and the path integral representation 
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FIG. 1: (a) A discrete jump process with both even (1 and 2) and odd (t and J,) variables, (b) jump process with only even 
variables (1, 1', 2, and 2'). The letters r, I, r' , I', ui, di, 112, d% (>0) are respective rates. 



([3]), we immediately obtain 



,0 / - f« Jfc,A(c)1(x(r))dr /x(«) ft) v = „,0 / -/* l 7[f,A](x(r))drv 
^ /n(0) ' ^ ' 



(A44) 



Because of J[c, A(c)]=J[l, A(l)] ([22]) is proved. 



Appendix B: A discrete jump model with both even and odd variables 



To show the unusual features of jump processes with both even and odd variables, we develop a simple model; sec 
Fig. (H])(a). Assuming that the variable i=l and 2 are even and t and 4- are odd, under time reversal (t—t—t) we have 

(i,t) = ft, 4-) and = ft,f). Although this model has eight rate parameters, they would not to be independent of 
each other if we required the model to have a steady-state solution satisfying the detailed balance principle. Using 
(|27 p -(|29 p and performing a simple calculation, we find these rates must obey the following three constraints: 



(A45) 
(A46) 
(A47) 

The first equation is a consequence of (f27|) or (|28|) . while the last two equations are from (|29|) . The equilibrium 
solutions can be easily obtained, which are 



Ir - I'r = 0, 
r — r + d\ — u\ = 0, 
I — I + d% — U2 = 0. 



Ptt,t)=P(U) = (* + 0/2(i + l' + r + r'), 
Pll t) =P^) = (r + r')/2(l + l' + r + r'). 

In addition, the total entropy production (|31|) is 



(A48) 
(A49) 



(J) = [<ftp(i,t) " u iP(i,l)} ln ! 1" [ rf 2P2,t) ~ u 2P(2,i)] In ; 



[rP(l,f) - ^(2,f)] ln 



P(i4) 
(r + r> (lit) 

ft + i')P(2,f) 



P(24) 



+ KP(i4) - l 'P(2,i)} In 



(r + r>(i4) 

ft + l')P(2,i) 



(A50) 



where P(<,t) and are the transient probabilities of the model at distinct states started from an initial distribution. 
Reader may easily check that, if p=p eq , the probability currents J or the terms before these logarithmic functions 
above are nonzero. 
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FIG. 2: (a) Time evolution of the probabilities at the four states in Fig. HI (a) and (b) the entropy production rates calculated by 
the classical Schnakenbergs formula (|A56[I and (|31[) proposed in this work. We choose £=ls _ , r=0.2s~ , £i=0.4s _ , ri=0.5s~ ; 
di=0.7s ; Mi=0.4s~ , d2=0.2s~ , and U2=0.8s~ , which satisfy (|A45[) - (lA47fl . The initial conditions are p^ ^=1.0, and the 
others vanish. 



It would be interesting to compare these results with those obtained in a conventional jump processes with only 
even variables, e.g. Fig. ©(b). Its states are respectively marked by the variables 1, 2, 1', and 2', all of which are 
even under time reversal. Because of p eq (i,t)=P eq («,^) in the model (a), here we additionally require p eq (i)=p eci (i'). 
Hence, we obtain three constraints on the rates given by 

d 1 -u 1 = 0, (A51) 
d 2 - u 2 = 0, (A52) 
lr' - I'r = 0, (A53) 



the equilibrium solutions 



p?=p'$ = l/2{r + l), (A54) 
P c 2 q =p^ = r/2(r + l) 7 (A55) 



and the classical total entropy production [3l| 



{J)c = (diPi - u x p v ) ln( dlPl ) + (d 2 p 2 - u 2 P2')\n(- dz 



uipv u 2 p 2 > 
+ (rpi - l P2 ) ln(^i) + (r'py - I'pt) In(^), (A56) 

lP2 Vpi> 

where Pi and p^ (i =1,2) are the transient probabilities of the model (b) at individual states started from an initial 
distribution. Obviously, in the model (a) if one did not take the odd variable into account and naively used (|A56|) . 
namely replacing pi and pv therein by P{%^) and respectively, he or she would find that the classical formula 

would not vanish as the system reaches the equilibrium states (|A48|) and (|A49|) . A numerical example to confirm 
these results is shown in Fig. ([2]). 
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